Open analytical ecosystems for digital humanities

Open science principles

• Emphasis on research process

• Transparency (data, methods, reporting)

• Reproducibility

• Openness (unlimited access and reuse)

• New modes of collaboration and initiatives

• Access to data is an institutional question. Using and tidying up the data is a research question

• Automation vs. point-n-click ?

Library catalogues: the data

https://github.com/rOpenGov/estc

• English Short Title Collection (ESTC) from British Library
• 1000 documents on history (~10% of the ESTC)
• Finland (Fennica), Europe (other catalogues), etc. etc.
• New use for old databases
• Quantitative knowledge production: book sizes, publishers, authors..

Publishing “history” in Britain and North America 1470-1800

Research questions

1. Who wrote history?
   
2. Where was it published?
3. How does the publishing of history change over the early modern period?

Load the data and tools

Load the data and tools in R:

load("df.RData")
library(bibliographica)

Fill missing dimensions

Estimate missing dimensions

kable(polish_dimensions("10 cm (12⁰)", fill = TRUE))

Author gender

Enriching data by external information

as.matrix(get_gender(polish_author(sample(unique(df$author.name), 20))$first)$gender)

##           [,1]  
## hugh      "male"
## william   "male"
## daniel    "male"
## edward    "male"
## john      "male"
## william   "male"
## gilbert   "male"
## samuel    "male"
## henry     "male"
## levi      "male"
## zachary   "male"
## alexander "male"
## john      "male"
## anthony   "male"
## philippe  "male"
## david     "male"
## robert    "male"
## edward    "male"
## calybute  NA    
## gilbert   "male"

Who wrote history?

• Authors (number of titles / paper use / life years)

• Times 1470 - 1800 ?

• Places: London, Ireland, Scotland, North America.. ?

• Language ?

• Gender ?

Who wrote history?

Top-10 authors (number of titles)

top_plot(df, "author.unique", 20)

Who wrote history?

Top-10 female authors (number of titles)

Who wrote history?

Title count vs. paper consumption

Document count vs. paper for top authors

ggplot(df2, aes(x = docs, y = paper)) + geom_text(aes(label = author.unique), size = 4)

Who wrote history?

Gender distribution for authors over time. Note that the name-gender mappings change over time. This has not been taken into account yet.

## 
## female   male 
##  0.037  0.963

Who wrote history?

Other questions to explore

• Publishing times 1470 - 1800 ?

• Places: London, Ireland, Scotland, North America.. ?

• Language ?

df2 <- df %>% filter(publication.place == "London")
df2 <- df %>% filter(language == "French")
df2 <- df %>% filter(publication.year >= 1700 & publication.year < 1800)
top_plot(df2, "author.unique", 10)

2. Where was history published ?

Top-10 places (number of titles)

top_plot(df, "publication.place", 10)

Where was history published ?

df2 <- df %>% filter(publication.country %in% c("France", "Germany")) %>%
    group_by(publication.decade, publication.country) %>%
    summarize(paper = sum(paper.consumption.km2, na.rm = TRUE), docs = n()) 
p <- ggplot(df2, aes(x = publication.decade, y = docs, color = publication.country)) +
     geom_point() + geom_smooth()
print(p)     

## Warning in loop_apply(n, do.ply): span too small. fewer data values than
## degrees of freedom.

## Warning in loop_apply(n, do.ply): pseudoinverse used at 1599

## Warning in loop_apply(n, do.ply): neighborhood radius 180.95

## Warning in loop_apply(n, do.ply): reciprocal condition number 0

## Warning in loop_apply(n, do.ply): There are other near singularities as
## well. 29224

## Warning in loop_apply(n, do.ply): span too small. fewer data values than
## degrees of freedom.

## Warning in loop_apply(n, do.ply): pseudoinverse used at 1599

## Warning in loop_apply(n, do.ply): neighborhood radius 180.95

## Warning in loop_apply(n, do.ply): reciprocal condition number 0

## Warning in loop_apply(n, do.ply): There are other near singularities as
## well. 29224

## Warning in loop_apply(n, do.ply): NaNs produced

Where was history published ?

Title count vs. paper

ggplot(df2,
     aes(x = log10(1 + docs), y = log10(1 + paper))) +
     geom_text(aes(label = publication.place), size = 3) +
     scale_x_log10() + scale_y_log10() 

Where was history published ?

Scotland, Ireland, US comparison:

df2 <- df %>%
    filter(!is.na(publication.country)) %>%
    group_by(publication.country) %>%
    summarize(paper = sum(paper.consumption.km2, na.rm = TRUE),
          docs = n()) %>%
    arrange(desc(docs)) %>%
    filter(publication.country %in% c("Scotland", "Ireland", "USA"))

Where was history published ?

p1 <- ggplot(subset(melt(df2), variable == "paper"), aes(y = value, x = publication.country)) + geom_bar(stat = "identity") + ylab("Paper consumption")
p2 <- ggplot(subset(melt(df2), variable == "docs"), aes(y = value, x = publication.country)) + geom_bar(stat = "identity") + ylab("Title count")
grid.arrange(p1, p2, nrow = 1)

3. How does the history publishing change in the early modern period ?

What can we say about the nature of the documents? Pamphlets (<32 pages) vs. Books (>120 pages) ? Book size statistics and development over time

Nature of the documents

Nature of the documents

Estimated paper consumption by document size

Nature of the documents

Document sizes over time

Serious statistical analysis (also in the Humanities)

• ~80 % of statistical analysis is tidying up of the data. Too often neglected and implicitly assumed by many tools. We provide new efficient tools also for this

• With open data principles, no need to reinvent the wheel for the same (or similar) datasets

• Things become stable. The research tool is corrected and perfected when it is transparent & potentially used also by others

• Possibilities of reuse with similar datasets is great

• Automatization allows reporting with minimal human intervention

Open science in (digital?) humanities

• Innovative use of computational and statistical methods

• New tools for old questions derived from the discipline itself

• Vast amounts of useful data not being shared or utilized

• Open access not enough. We need open sharing of research data and methods to study “traditional” questions

Barriers to open science in the humanities

• Institutions that hold the raw data are reluctant to give full access to data (even to researchers of the same institution). Why?

• Research process is not opened and research data is not shared in the Humanities. Transparency, reproduction, collaboration, new initiatives are missing. Why?

• Short answer: Cultural change takes time. We need concrete examples in the core field of the Humanities that actually prove OPEN DATA PRINCIPLES as useful.

• Statistical software for computational social sciences & humanities
• Open source / Fully transparent
• Reproducible workflows
• Preprocessing, enrichment, integration, analysis, visualization, reporting..
• Strong developer community (Finland + International)
• Practical & low-cost research tools rather than polished software product
• Based on analogous ecosystems from other fields (bioinformatics, ecology..)